If someone went up to some random person of colour on the street and said 'get back to where you came from!', well, that's obviously very different.
So if I go out and find 4 women who have the same exact background as these four congresswomen (women of color) walking on the street and say "Hey, go back to where you came from" <== that is racist/racially charged language/behavior.
However in the correct context (I am having some sort of row with them) it's not racist.
So how about this:
I work with 4 women who have the same exact background as these four congresswomen (women of color) at a major telecommunications company. I get into a major fight about some company policy or procedure. One of the women even calls me a mf-er in front of everyone at work.
I then get up and say: "Why don't these women go back and help fix the totally broken telecommunications system in the placed from which they came?" <== Would saying this be racist? Because if I had ever said anything like this at work, at the very least I would have been suspended, if not fired.